Find out the IP addresses if appeared more than 4times

blason · 07-15-2019, 05:05 AM

Hi Guys,

I have huge list of IP addresses and I need to find/filter out the IP addresses which are appeared more than 4 times.

I am looking for awk or may be bash tricks?

can this be achieved?

TIA
Blason R

danielbmartin · 07-15-2019, 05:23 AM

Help us to help you. Provide a sample input file (10-15 lines will do). Construct a sample output file which corresponds to your sample input and post both samples here. With "InFile" and "OutFile" examples we can better understand your needs and also judge if our proposed solution fills those needs.

Daniel B. Martin

pan64 · 07-15-2019, 05:50 AM

did you try uniq -c (for example?)
yes, awk can be useful too.

blason · 07-15-2019, 05:56 AM

cat /var/log/maillog | grep "after AUTH from" | awk '{print $NF}' | sed -e 's/\[/ /g' -e 's/\]//g' | awk '{print $NF}' | sort

This is sample input file
49.67.70.232
121.226.70.175
114.232.72.113
121.226.59.152
45.13.39.115
117.86.91.120
114.232.193.25
121.232.8.169
121.226.126.140
114.231.140.157
114.232.254.54
111.227.162.254
114.231.60.147
114.232.43.54
114.232.72.48
114.231.140.157
114.232.123.154
180.121.176.227
121.226.61.222
180.121.199.244
121.232.126.120
49.67.64.22
111.227.162.254
114.231.140.157
180.120.94.86
78.128.113.67
121.232.17.93
114.232.193.9
114.232.254.72
180.121.199.168
114.232.201.249
180.120.77.17
114.232.254.54
121.226.56.108
180.121.188.167
114.232.218.128
111.227.162.254

*******************
This is I am getting
285 103.125.191.84
1328 103.234.97.254
84 106.208.135.163
2 106.57.150.244
2 109.169.11.211
143 111.227.162.254
10 114.106.151.43
6 114.229.21.244
15 114.231.136.105
31 114.231.136.120
38 114.231.136.132
8 114.231.136.133
25 114.231.136.168
6 114.231.136.200
10 114.231.136.202
8 114.231.136.231
21 114.231.136.241
8 114.231.136.245
36 114.231.136.247
12 114.231.136.39
11 114.231.136.46
14 114.231.136.47
8 114.231.136.77
8 114.231.137.130
7 114.231.137.142
7 114.231.137.153
10 114.231.137.160
31 114.231.137.166
3 114.231.137.187
54 114.231.137.197
4 114.231.137.204

here I am removing entries only appearing once.

but I need to find out the entries which are appearing more than 4 times and exclude those are below 4.

pan64 · 07-15-2019, 06:09 AM

ok, since awk is already involved:
1. do not use cat|grep|awk|sed|awk|sort chain, because it can be solved with a single awk
2. would be nice to see some lines of that log file

you can start with something like this:

Code:

awk -F']' '/after AUTH from/ { ip[$NF]++ }
           END { for (i in ip) print "ip=" i ", ("ip[i]")" }'

not tested

blason · 07-15-2019, 06:34 AM

Here are

Jul 14 17:31:31 mail postfix/smtpd[26111]: lost connection after AUTH from unknown[180.120.191.39]
Jul 14 17:31:34 mail postfix/smtpd[1331]: lost connection after AUTH from unknown[180.125.131.77]
Jul 14 17:31:46 mail postfix/smtps/smtpd[25242]: lost connection after AUTH from unknown[45.13.39.115]
Jul 14 17:31:47 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[180.121.150.182]
Jul 14 17:31:54 mail postfix/smtpd[26111]: lost connection after AUTH from unknown[121.232.8.7]
Jul 14 17:31:59 mail postfix/smtpd[1331]: lost connection after AUTH from unknown[114.232.217.208]
Jul 14 17:32:07 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[114.232.59.9]
Jul 14 17:32:13 mail postfix/smtpd[26111]: lost connection after AUTH from unknown[180.121.150.182]
Jul 14 17:32:15 mail postfix/smtpd[1331]: lost connection after AUTH from unknown[49.79.137.34]
Jul 14 17:32:27 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[114.231.137.42]
Jul 14 17:32:36 mail postfix/smtps/smtpd[25242]: lost connection after AUTH from unknown[185.211.245.198]
Jul 14 17:32:44 mail postfix/smtpd[1331]: lost connection after AUTH from unknown[117.86.178.209]
Jul 14 17:32:47 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[180.121.150.182]
Jul 14 17:32:50 mail postfix/smtpd[26111]: lost connection after AUTH from unknown[49.67.164.187]
Jul 14 17:32:50 mail postfix/smtps/smtpd[27984]: lost connection after AUTH from unknown[185.211.245.198]
Jul 14 17:33:06 mail postfix/smtpd[1331]: lost connection after AUTH from unknown[121.232.17.128]
Jul 14 17:33:09 mail postfix/smtpd[26111]: lost connection after AUTH from unknown[180.121.141.252]
Jul 14 17:33:16 mail postfix/smtpd[1331]: lost connection after AUTH from unknown[121.226.92.32]
Jul 14 17:33:44 mail postfix/smtps/smtpd[28128]: lost connection after AUTH from unknown[45.13.39.115]
Jul 14 17:35:44 mail postfix/smtps/smtpd[25242]: lost connection after AUTH from unknown[45.13.39.115]
Jul 14 17:37:40 mail postfix/smtps/smtpd[28128]: lost connection after AUTH from unknown[45.13.39.115]
Jul 14 17:39:06 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[49.67.111.97]
Jul 14 17:39:06 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[49.67.64.36]
Jul 14 17:39:08 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[121.226.63.59]
Jul 14 17:39:10 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[114.232.217.184]
Jul 14 17:39:12 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[49.67.67.223]
Jul 14 17:39:12 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[121.232.16.38]
Jul 14 17:39:14 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[121.226.92.162]
Jul 14 17:39:15 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[180.125.131.49]
Jul 14 17:39:17 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[121.226.60.137]
Jul 14 17:39:18 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[114.231.141.167]
Jul 14 17:39:20 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[49.67.141.195]
Jul 14 17:39:21 mail postfix/smtps/smtpd[28128]: lost connection after AUTH from unknown[185.211.245.198]
Jul 14 17:39:22 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[180.121.148.113]
Jul 14 17:39:22 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[49.67.70.133]
Jul 14 17:39:23 mail postfix/smtpd[7126]: lost connection after AUTH from unknown[121.226.126.159]
Jul 14 17:39:24 mail postfix/smtpd[26115]: lost connection after AUTH from unknown[121.232.19.144]

pan64 · 07-15-2019, 06:45 AM

Code:

awk -F'[][]' '/after AUTH from/ { ip[$(NF-1)]++ }
           END { for (i in ip) print "ip=" i ", ("ip[i]")" }'

this will work, you need only to add the include/exclude filter

rnturn · 07-15-2019, 12:56 PM

Quote:

Originally Posted by blason

Hi Guys,

I have huge list of IP addresses and I need to find/filter out the IP addresses which are appeared more than 4 times.

I am looking for awk or may be bash tricks?

can this be achieved?

TIA
Blason R

Try:

Code:

sort < ip_addr.lis | uniq -c | grep -v -E ' [1-3] '

This looks for and ignores the 'uniq -c' counts less than 4. (The spaces in the grep pattern are important.)

(Note: not thoroughly tested as I don't have any long lists of stuff I can run through 'uniq' at the moment.)

HTH...

MadeInGermany · 07-15-2019, 02:04 PM

Quote:

Originally Posted by pan64

Code:

awk -F'[][]' '/after AUTH from/ { ip[$(NF-1)]++ }
           END { for (i in ip) print "ip=" i ", ("ip[i]")" }'

this will work, you need only to add the include/exclude filter

Add your condition like this.

Code:

awk -F'[][]' '/after AUTH from/ { ip[$(NF-1)]++ } END { for (i in ip) if (ip[i]>=4) print i,"("ip[i]")" }' /var/log/maillog

Explanations:
The -F is a character set of [ and ] so the 2nd last field $(NF-1) holds the ip.
Store a count in the array ip that is addressed by the $(NF-1).
At the END loop through the array and print if the condition is met.

blason · 07-15-2019, 10:22 PM

Nice ideas folks.

let me give a try and see which one actually works

Thanks again.